Develop#21
Merged
Merged
Conversation
Why: - Indexing against Ollama WordPiece-based embedding models can exceed model context limits because chunk sizing used only cl100k_base token counts. - Ollama error payloads using "input length exceeds the context length" were not consistently classified as token-limit failures. What: - Added tokenizer strategies (tiktoken, character, simple) and strategy-based default tokenizer factory. - Added ACI_TOKENIZER config/env support and wired tokenizer selection into service initialization so chunking and summary generation share the chosen tokenizer. - Expanded token-limit error detection to include context-length style 400 responses. - Updated unit/property tests for tokenizer strategy selection, offline-safe tokenizer tests, config strategy generation, and token-limit message coverage. Test: - uv run pytest tests/unit/test_tokenizer.py tests/property/test_embedding_client_properties.py tests/property/test_config_properties.py -q (pass) - uv run ruff check src tests (pass) - uv run pytest tests/ -v --tb=short -q --durations=10 --maxfail=1 (fails in this environment due proxy blocking tiktoken encoding download) - uv run mypy src --ignore-missing-imports --no-error-summary (existing repo-wide mypy errors)
Why: MCP runtime path mappings were bypassed for absolute host paths on Windows, which broke the new MCP path-resolution flow and caused pytest failures. The latest MCP changes also expanded the context contract and exposed a flaky Hypothesis deadline in file scanner tests. What: apply runtime path mappings before native absolute-path fallback, add runtime path resolution coverage for Windows and POSIX host paths, update MCP/qdrant/property tests to match the new contract, and disable the flaky deadline on the ignore-pattern property test. Test: uv run ruff check src tests (passed) Test: uv run pytest tests/ -v --tb=short -q --durations=10 (passed, 660 passed / 16 skipped)
…ch-with-ollama-models fix(tokenizer): add configurable tokenizer strategies and improve token-limit detection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.